The Internet comprises a vast number of computers and computer networks that are interconnected through communication links, with information being exchanged using various services such as electronic mail and the World Wide Web (also referred to as the “Web”). In addition to providing access to information, the Web has increasingly become a medium that is used to search for, shop for and order items (such as products, services and/or information) that are for purchase, rent, lease, license, trade, evaluation, sampling, subscription to, etc. In many circumstances, a user can visit the Web site of a Web merchant (or a “Web store”) or otherwise interact with an online retailer or electronic marketplace that provides one or more items, such as to view information about the items, give an instruction to place an order for one or more items, and provide information needed to complete the purchase (e.g., payment and shipping information). The Web merchant then fulfills the order by providing the ordered items to the indicated recipient, such as by providing product items that have been ordered through physical distribution channels (e.g., shipment via a governmental postal service or private common carrier) or electronically (e.g., via download over the Internet, such as for digital music or videos) as appropriate. Ordered service items may similarly be provided electronically (e.g., providing email service) or physically (e.g., performing cleaning services at the purchaser's house).
In addition to such Web-based interactions between computers (e.g., interactions initiated by users using Web browser applications to interactively request Web pages from Web servers), various distributed computing systems are known in which multiple computer systems interact in other manners in order to achieve a goal. For example it is often desirable for an application program on a computer system to interact with remote systems and applications in order to obtain various types of information and functionality that are not part of the application program. By performing such interactions, an application program may be able to leverage information and functionality from vast numbers of other computer systems over the Internet or other networks. In order to enable such interactions between remote computer systems and application programs, various programmatic interaction mechanisms have been developed. For example, remote procedure call (“RPC”) protocols have long existed that allow a program on one computer to cause a program on another computer to be executed, and various object-oriented and other architectures such as CORBA (“Common Object Request Broker Architecture”), Java RMI (“Remote Method Invocation”), JavaSpaces, Jini, JXTA, UPnP (“Universal Plug and Play”) and DCOM (“Distributed Component Object Model”) provide similar capabilities. In addition, a variety of middleware programs have been implemented to connect separate applications (often of distinct types and from unrelated sources) to allow communication. For example, various EDI (“Electronic Data Interchange”) networks exist that provide standard mechanisms to allow a computer system of one user of the network to send data to a computer system of another user of the network.
The widespread popularity of the Web has also provided additional opportunities for computers to inter-communicate in various programmatic manners. For example, there is growing use of the Web to provide so-called “Web services,” which typically involve the programmatic interaction of remote applications to exchange information via defined APIs (“application program interfaces”), or the like. Web services may allow heterogeneous applications and computers to interact, and can be defined and implemented using a variety of underlying protocols and techniques. For example, some Web service implementations return data in XML (“eXtensible Markup Language”) format using HTTP (“HyperText Transport Protocol”) in response to a Web service invocation request specified as a URI (“Uniform Resource Identifier”), such as a URL (“Uniform Resource Locator”) that includes a specified operation and one or more query parameters. In other implementations, additional underlying protocols are used for various purposes, such as SOAP (“Simple Object Access Protocol”) for standard message exchange, WSDL (“Web Services Description Language”) for description of service invocations, and UDDI (“Universal Description, Discovery, and Integration service”) for discovery of available services.
Unfortunately, while Web services and other programmatic interaction mechanisms allow various application programs and computers to interact, such interactions are typically limited in various ways. For example, the types of information and functionality that are available to be requested using such programmatic interactions are typically restricted to very limited types of requests that the remote computer systems and applications can automatically fulfill (e.g., to provide a specified predefined group of information, such as a Web page or file, or to perform a specified database query on a specified database). Moreover, there is a very large class of tasks which computers and application programs cannot easily automatically perform, but which humans can typically easily perform, referred to herein as “human performance tasks.” This is due at least in part to various cognitive and other mental capabilities of humans that are not easily encoded in automated programs, such as the ability to use human judgment to form opinions, to perform abstract or common-sense reasoning, to perform various discernment and perception tasks (e.g., visual and aural pattern recognition, such as based on experience), to use cultural awareness and emotional intelligence, and to perform various other everyday yet highly complex kinds of perception, cognition, reasoning and thinking.
However, existing settings that allow parties to make tasks available for performance by human users also have various problems. One example of a problem that arises in the context of human performance of tasks is that of efficiently and effectively determining the quality and other acceptability of the results from the performance of the tasks. In some cases, to ensure that results of task performance by others are acceptable, the parties that provide the tasks may perform some or all of the provided tasks themselves in order to evaluate the others' results, which may assist in ensuring quality of results but also typically causes significant amounts of time and energy to be used by the task providing parties. In other cases, parties who provide tasks may merely use the results from others without any verification that the results are actually acceptable (such as based solely on the task performers' qualifications and/or prior activities), which may cause inaccurate or otherwise inappropriate results to be used.